Multi Team: Automatically create and assign team default pools by SameerMesiah97 · Pull Request #69768 · apache/airflow

SameerMesiah97 · 2026-07-11T21:43:03Z

Description

This change introduces automatic default pool management for multi-team deployments.

With this implementation, a corresponding default pool named default_pool_<team_name> is automatically created on team creation. During DAG parsing, tasks that do not explicitly specify a pool are assigned to their team's default pool based on the DAG bundle association, while tasks with explicitly configured pools are left unchanged.

The implementation also extends the team management commands so that teams sync reconciles missing default team pools for existing teams, and teams delete automatically removes the system-managed default team pool.

Rationale

In multi-team deployments, tasks without an explicit pool are currently assigned to the global default_pool, allowing one team's workload to consume capacity shared across all teams. Avoiding this requires users to manually create and configure team-specific pools and update every task to reference them.

By automatically provisioning a per-team default pool and applying it during DAG parsing, Airflow provides sensible defaults while preserving explicit pool assignments. This reduces operational overhead and helps isolate execution capacity between teams without requiring changes to existing DAGs.

Tests

Added unit tests verifying that:

Tasks using default_pool are assigned to the appropriate team default pool during DAG parsing.
Tasks with explicitly configured pools are not modified.
DAGs without an associated team continue to use default_pool.
Creating a team automatically provisions its corresponding default pool.
teams sync recreates missing default team pools for existing teams.
Deleting a team automatically removes its default team pool.
Team default pool names are generated correctly.

Documentation

Updated the multi-team documentation to describe automatic creation of team default pools and how they are applied during DAG parsing.

Backwards Compatibility

Existing behaviour is preserved when multi-team mode is disabled. Tasks with explicitly configured pools continue to use their configured pool without modification.

Closes: #69170

SameerMesiah97 · 2026-07-18T22:17:25Z

@o-nikolas @ferruzzi

Tagging you here as this is your domain i.e. multi-team.

vincbeck · 2026-07-20T14:01:38Z

+    ):
        associations.append(f"{pool_count} pool(s)")

+    default_pool = session.scalar(select(Pool).where(Pool.pool == Pool.get_default_team_pool_name(team.name)))


That's an edge case but could be named default_pool_marketing and be global. We should check that the pool is also associated to the team

And additional check has been added to the where clause to ensure that the select pool is associated with the team.

o-nikolas

Thanks for the contribution! The code needs a fair amount of work but I like the idea!

o-nikolas · 2026-07-23T01:15:24Z

    The specified team must exist in the database (create it first with ``airflow teams create``).

+    When ``core.multi_team`` is enabled, ``airflow teams create`` automatically
+    creates a default pool named ``default_pool_<team_name>``. Tasks in DAG


Suggested change

creates a default pool named ``default_pool_<team_name>``. Tasks in DAG

creates a default pool named ``default_pool_<team_name>``. Tasks in Dag

Should it not be DAG instead of Dag as it is an acronym?

Edit: Nvm, it appears that 'Dag' is the convention in the docs.

o-nikolas · 2026-07-23T01:15:43Z


+    When ``core.multi_team`` is enabled, ``airflow teams create`` automatically
+    creates a default pool named ``default_pool_<team_name>``. Tasks in DAG
+    bundles associated with that team that do not explicitly specify a pool will


Suggested change

bundles associated with that team that do not explicitly specify a pool will

bundles associated with that team which do not explicitly specify a pool will

I am not sure if 'which' is more gramatically correct than 'that' here as the part after is necessary for the sentence to communicate the complete idea. Regardless, I have rephrased this part for clarity.

o-nikolas · 2026-07-23T01:16:53Z

+    bundles associated with that team that do not explicitly specify a pool will
+    use the team's default pool automatically.
+
+    Existing multi-team deployments should run ``airflow teams sync`` after upgrading


Multi-team is in experimental phase right now. I'm not sure we want to a very time based piece in our docs. Update the news fragment to call out this guidance about syncing existing multi-team deployments.

o-nikolas · 2026-07-23T01:24:08Z

+    default_pool = session.scalar(select(Pool).where(Pool.pool == Pool.get_default_team_pool_name(team.name)))
+
+    if default_pool:
+        session.delete(default_pool)


This delete is happening way too soon. There is a check below "are you sure you want to delete..." you should not delete anything until after that.

This has been moved down to the try/except block where team deletion is attempted.

o-nikolas · 2026-07-23T01:25:39Z

+                    slots=conf.getint(
+                        "core",
+                        "default_pool_task_slot_count",
+                    ),


This clobbers any specific pool slots admins may have set if the pool already exists. You really just want create if missing not create or update

We are now querying for an existing default pool and creating it only if no pool is found.

o-nikolas · 2026-07-23T01:28:11Z

+        with conf_vars(
+            {
+                ("core", "multi_team"): "True",
+                ("multi_team", "default_pool_task_slot_count"): "128",


Suggested change

("multi_team", "default_pool_task_slot_count"): "128",

("core", "default_pool_task_slot_count"): "111",

This is a core config, and the default is already 128, setting it to something else ensures that this is working in your test.

assign tasks without an explicit pool to the team's default pool during DAG parsing. Add unit tests covering pool assignment and team creation, and update the multi-team documentation to describe the new behaviour.

…missing pools for existing teams. Automatically remove the system-managed default team pool during while preserving protection for user-created team pools. Add tests covering reconciliation and cleanup.

…ct config key in test.

Only create missing default team pools during teams sync, improve cleanup during team deletion, and update the documentation and tests.

boring-cyborg Bot added area:CLI area:DAG-processing kind:documentation labels Jul 11, 2026

SameerMesiah97 marked this pull request as ready for review July 12, 2026 13:12

SameerMesiah97 requested review from XD-DENG, ashb, bugraoz93, dheerajturaga, ephraimbuddy, henry3260, jedcunningham and potiuk as code owners July 12, 2026 13:12

SameerMesiah97 mentioned this pull request Jul 12, 2026

Multi Team: Add team default pools for tasks #69170

Open

2 tasks

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from 4c57afb to 9691842 Compare July 12, 2026 15:03

justinpakzad reviewed Jul 12, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/dag_processing/dagbag.py

SameerMesiah97 marked this pull request as draft July 12, 2026 16:42

stephen-bracken reviewed Jul 13, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/models/pool.py

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from 9691842 to fdcf3d6 Compare July 18, 2026 19:18

SameerMesiah97 marked this pull request as ready for review July 18, 2026 20:05

justinpakzad reviewed Jul 18, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/cli/commands/test_team_command.py Outdated

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from fdcf3d6 to 9572e56 Compare July 18, 2026 21:54

stephen-bracken reviewed Jul 19, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/cli/commands/team_command.py Outdated

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 20, 2026

vincbeck reviewed Jul 20, 2026

View reviewed changes

o-nikolas reviewed Jul 23, 2026

View reviewed changes

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch 2 times, most recently from 6caec49 to 4ff010e Compare July 24, 2026 00:40

SameerMesiah97 marked this pull request as draft July 24, 2026 08:41

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from 4ff010e to c4de299 Compare July 24, 2026 19:03

Sameer Mesiah added 3 commits July 24, 2026 21:15

Reconcile default team pools during teams sync, including recreating …

6f60e75

…missing pools for existing teams. Automatically remove the system-managed default team pool during while preserving protection for user-created team pools. Add tests covering reconciliation and cleanup.

Update documentation to instruct pro-active team sync and fix incorre…

1a9a37c

…ct config key in test.

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from c4de299 to 88dc14b Compare July 24, 2026 20:16

Refine team default pool management

1ca0fb1

Only create missing default team pools during teams sync, improve cleanup during team deletion, and update the documentation and tests.

SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from 88dc14b to 1ca0fb1 Compare July 24, 2026 20:19

SameerMesiah97 marked this pull request as ready for review July 24, 2026 21:35

SameerMesiah97 requested review from o-nikolas, stephen-bracken and vincbeck July 25, 2026 16:17

	creates a default pool named ``default_pool_<team_name>``. Tasks in DAG
	creates a default pool named ``default_pool_<team_name>``. Tasks in Dag

	bundles associated with that team that do not explicitly specify a pool will
	bundles associated with that team which do not explicitly specify a pool will

	("multi_team", "default_pool_task_slot_count"): "128",
	("core", "default_pool_task_slot_count"): "111",

Uh oh!

Conversation

SameerMesiah97 commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SameerMesiah97 commented Jul 18, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SameerMesiah97 Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

SameerMesiah97 commented Jul 11, 2026 •

edited

Loading

SameerMesiah97 Jul 23, 2026 •

edited

Loading